Apparatus and method for vision processing on network based intelligent service robot system and the system using the same

ABSTRACT

There are provided an apparatus and method for vision processing on a network based intelligent service robot system and a system using the same. A robot can move to a target object, avoiding obstacles without helps of a robot server interfaced with a robot terminal over network, by extracting/processing three-dimensional distance information of external objects, using a stereo camera, a low price image processing dedicated chip and an embedded processor. Therefore, the intelligent robot can travel and move using only a stereo camera image processing without other sensors, and further provides users with various functional services with low expense.

TECHNICAL FIELD

The present invention relates to a vision processing on a network basedintelligent service robot, and more particularly, to a technology that astereo matching result for an acquired stereo image is used for arobot's traveling and moving in the network based intelligent servicerobot.

BACKGROUND ART

Most of conventional vision processing apparatuses in a robot system isrestrictively used in recognizing an external circumstance of a robotthrough input information of only one camera. Therefore, there is adisadvantage that it is necessary to compositively use informationobtained from other sensors such as a ultrasonic sensor, an infraredsensor, or the like, in order to make use of the conventional visionprocessing apparatuses for the travel and movement of the robot. Inparticular, because a necessary vision processing operation is performedin a robot server connected to a robot terminal over network in thenetwork based intelligent service robot, there is also a disadvantagethat it causes excessive communication traffics between the robotterminal and the robot server.

FIG. 1 is a block diagram illustrating a whole system of a generalnetwork based intelligent service robot. As illustrated in FIG. 1, thegeneral network based intelligent service robot system includes a robotserver 20 and a plurality of robot terminals 10: 1, 2, . . . , ninterfaced with the robot server 20. Herein, each of the robot terminals10 is configured with a robot vision processing unit 101 for acquiringand processing an external image, a robot control unit 102 forcontrolling the robot terminal 10, a robot server communication unit 103for serving a role of communicating with the robot server 20, and arobot sensor and drive unit 104 for sensing various kinds of externalcircumstances and driving the robot terminal 10.

In order to drive the robot terminal 10 in the above system, the robotterminal 10 transmits acquired image information to the robot server 20,and the robot server 20 controls the travel of the robot afterrecognizing/processing images for obstacles to the robot's traveling andmoving.

In the network based intelligent service robot system, since complexapplications with high capacity and a load requiring high speedoperation, which are difficult to be processed in each robot terminal10, are concentrated on the robot server 20 connected over the network,it is possible to implement the robot terminal with a relative low cost.Accordingly, it is possible to provide users with various and highquality services with low price.

That is, in order to provide the users with various functional serviceswith low price in the network based intelligent service robot, the robotcontroller 102 of the robot terminal 10 is implemented by using a lowpower embedded processor having advantages in an aspect of price andpower consumption, instead of using a personal computer with high price,considering the robot terminal 10 to become cheap.

In addition, the robot controller 102 of the robot terminal 10 isimplemented such that it has relatively low computing power.

Furthermore, in case of processing the complex applications in the robotserver 20, although the robot terminal 10 may be implemented at lowprice relatively, communication traffic with the robot server 20increases after all to raise the communication expense because lots ofapplications for executing the service are performed within the robotserver 20. On the contrary, if various functions are performed in therobot terminal 10 for reducing the expense of the communication with therobot server 20, the load to be processed in the robot terminal 10 mayincrease so that the robot controller 102 should be implemented at highprice for having high computing power.

Thus, there is a need to make a compromise with the constitution of therobot terminal 10 and the communication traffic with the robot server20, when configuring the network based intelligent service robot system.In particular, the communication traffic between the robot server 20 andthe robot terminal 10, becomes a factor which has a large effect onstability of the system as well as the communication expense because theplurality of the robot terminals 10 are connected to the one robotterminal 20 as illustrated in FIG. 1.

Therefore, in order that the network based intelligent service robot mayprovide the users with the various functional services with low price,the cost of the robot terminal should be reduced to be low, first. Inaddition, another consideration is the communication expense which iscaused by using the network. That is, according to the prior art, thereis a drawback that it should be considered the communication trafficbetween the robot terminal and the robot server in the network basedintelligent service robot, which is a cause for a considerableadditional expense due to the network communication.

DISCLOSURE Technical Problem

It is an object of the present invention to provide an apparatus and amethod for vision processing on a network based intelligent servicerobot in which an image data, which occupies most of communicationtraffic with a robot server, is processed in a robot terminal itself,while not implementing the robot terminal with high cost, and a systemusing the same.

It is another object of the present invention to provide an apparatusfor vision processing on a network based intelligent service robot inwhich the apparatus for vision processing is implemented using a stereocamera, and an acquired data through the stereo camera enables the robotto travel and move without helps of other sensors byextracting/processing three-dimensional distance information with regardto external objects using a stereo matching technology as well as forrecognizing an external circumstance of the robot, a vision processingmethod thereof, and a system using the same.

It is a further another object of the present invention to provide anapparatus and a method for vision processing on a network basedintelligent service robot capable of reducing a network communicationcharge between a robot terminal and a robot server by positioning theaforementioned vision processing apparatus in the robot terminal and nottransmitting the image data to the robot server, and a system using thesame.

Technical Solution

In one aspect of the present invention, there is provided an apparatusfor vision processing on a network based intelligent service robotincluding: a stereo camera unit acquiring images from two cameras ofleft/right ones incorporated therein; an input image pre-processing unitfor calibrating characteristic differences between the two cameras withrespect to the left/right images inputted through the stereo cameraunit; a stereo matching unit for performing a stereo matching bycomputing a disparity map through the stereo matching over theleft/right images inputted through the input image pre-processing unit;an image post-processing unit for extracting a depth map for the imageoutputted from the stereo matching unit, and performing a segmentationfor segmenting different objects from the extracted depth map; and animage output selection unit for selectively outputting the output imageof the respective units.

In another aspect of the present invention, there is provided a methodfor vision processing on a network based intelligent service robot, themethod including: acquiring an input image from two cameras; performingan input image pre-processing to calibrate the left/right image signalsacquired from the two cameras to have the same circumstance, usingvarious image processing methods; performing a stereo matching, at astereo matching unit, by computing a disparity map after finding outpoints corresponding to each other in the left/right images calibratedin an input image pre-processing unit; extracting a depth map bycalculating a depth based on the disparity map computed at the stereomatching unit, and performing an image post-processing for segmentingdifferent objects by the extracted depth map; extracting the horizontallength, the vertical length, and the distance of the segmented objectfrom the robot terminal, and controlling the robot to travel and move bysegmenting/recognizing an object to track or an obstacle on the basis ofthis extraction result.

In a further another aspect of the present invention, there is provideda system using an apparatus for vision processing on a network basedintelligent service robot, including: a robot terminal including avision processing unit of a robot having the apparatus claimed in claim1 for acquiring an external image, a robot control unit for controllinga whole robot terminal, a robot server communication unit for serving arole of communicating with a robot server, and a robot sensor/drive unitfor sensing an external circumstance and driving the robot; and therobot server interfaced with the robot terminal over network, forcontrolling the robot under predetermined circumstances.

It is to be understood that both the foregoing general description andthe following detailed description of the present invention areexemplary and explanatory and are intended to provide furtherexplanation of the invention as claimed.

Advantageous Effects

According to the present invention, the information required for therobot's traveling and moving is processed in the robot terminal itselfwithout transmitting the information to the robot server by configuringthe robot terminal with the stereo camera, and a low price dedicatedchip and an embedded processor. Therefore, it is possible to reduce thecommunication expense which may occur in case of transmitting thenecessary image information for driving the intelligent service robot,so that the present invention is effective for embodying the networkbased intelligent service robot capable of providing users with variousfunctional services with low cost.

Furthermore, according to the present invention, since the acquiredimage can be used for extracting the three-dimensional distanceinformation of the external objects from the stereo camera as well asrecognizing the external circumstance by implementing the visionprocessing apparatus of the intelligent robot using the stereo camera,the present invention is effective for performing the intelligent robotto travel and move only with the stereo camera image processing withoutother sensors.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a whole system of a generalnetwork based intelligent service robot;

FIG. 2 is a block diagram illustrating an internal constitution of avision processing unit 101 of a network based intelligent service robotterminal 10 according to an embodiment of the present invention;

FIG. 3 is a photograph of a resultant image illustrating a visionprocessing unit of an intelligent service robot according to the presentinvention; and

FIG. 4 is a flow chart illustrating a vision processing operation of anintelligent service robot according to the present invention.

BEST MODE

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings.

FIG. 2 is a block diagram illustrating an internal constitution of avision processing unit 101 of a network based intelligent service robotterminal 10 according to an embodiment of the present invention.

The vision processing unit 101 of the network based intelligent servicerobot terminal of the present invention includes a stereo camera unit1010, an input image pre-processing unit 1011, a stereo matching unit1012, and an image post-processing unit 1013.

By means of the vision processing unit 101 in the robot terminal,information needed for the travel and movement of the robot is nottransmitted to the robot server 20, but can be processed through a robotcontroller 102 configured with a low price dedicated chip and anembedded processor in the robot terminal itself.

Considering each element of the vision processing unit 101 in detail,the stereo camera 1010 acquires images from two cameras, i.e., a leftone and a right one. Herein, the acquired images are output images ofthe camera, which are represented in FIG. 3( a).

The input image pre-processing unit 1011 employs various kinds of imageprocessing methods in order that the stereo matching unit 1012 mayeasily perform stereo matching with respect to the input image of thestereo camera 1010. The various kinds of image processing methodsinclude camera calibration, scale-down of the image, rectification ofthe image, brightness control of the image, etc. The noise existing inthe image is removed using this image processing method. Furthermore, incase that there exits a difference in brightness level or contrastbetween the image inputted through the two cameras, the image signalsinputted from the two cameras are calibrated to be equal to each otherby equalizing this image information using such image processingmethods. In addition, a histogram equalization, an edge detection or thelike is performed over the images outputted from the two cameras so thatthe images of which the function is wholly improved are outputted aftercalibration. This outputted image of the input image pre-processing unit1011 is shown in FIG. 3( b).

The stereo matching unit 1012 finds out points corresponding to eachother in the left/right images calibrated at the input imagepre-processing unit 1011 and computes a disparity map. In virtue of thestereo matching operation using this disparity map, the output image ofthe stereo matching unit 1012 is achieved such that the distanceinformation of the object is represented as a bright color for a closeobject and a dark color for a distant object, respectively, asillustrated in FIG. 3( c).

The image post-processing unit 1013 extracts a depth map by calculatinga depth based on the disparity map computed at the stereo matching unit1012. Then, a segmentation work for different objects is performed fromthe extracted depth map to obtain the output image of the imagepost-processing unit 1013, as shown in FIG. 3( d). The imagepost-processing unit 1013 extracts the horizontal length, the verticallengths of the segmented objects, and the distances of the segmentedobjects from the robot terminal, and performs an operation fordiscriminating whether the objects are objects to track or obstacles onthe basis of the extraction results. By using the results of the imagepost-processing unit 1013, the robot control unit 102 can control therobot to travel and move in the robot terminal itself without beingconnected to the robot server over network.

An image output selection unit 1014 selects the output image of thevision processing unit 101 of the intelligent service robot. That is,the image output selection unit 1014 may select anyone of the imagesacquired through the left/right cameras, the resultant image of theinput image pre-processing unit 1011, the resultant image of the stereomatching unit 102, and the resultant image of the image post-processingunit 1013.

By implementing the vision processing unit 101 of the network basedintelligent robot terminal as illustrated in FIG. 2, it is possible toextract the three-dimensional information of external objects from theimage captured at the stereo camera. Thus, there is an advantageousmerit that it is possible to perform the intelligent robot to travel andmove only with the stereo camera image process without other sensors. Inaddition, since it is not necessary to transmit the image data, which isa main cause for the communication traffic, to the robot server, thenetwork traffic with the robot server is remarkably reduced.Accordingly, there are advantages that the communication expensesconsumed for network communication may be reduced, and further thenetwork based intelligent service robot system in which a plurality ofrobot terminals are interfaced with one another becomes stable.

A method for vision processing on the network based intelligent servicerobot according to the present invention will be set forth herebelowwith reference to FIG. 4. FIG. 4 is a flow chart illustrating anoperation of the vision processing unit 101 of the intelligent servicerobot terminal.

To begin with, the robot terminal is changed its direction toward theobject by operating the camera pan/tilt through a sound source tracking(S1). The input image is acquired through the stereo camera unit 1011(S2).

An input image pre-processing is performed for adjusting the left/rightimage signals acquired through the two cameras of the stereo camera unit1011 to have the same circumstances using various image processingmethods (S3).

A stereo matching is performed for obtaining the image that isrepresented as a bright color and a dark color for a close object and adistant object, respectively by computing the disparity map afterfinding out points corresponding to each other in the left/right imagescalibrated at the input image pre-processing unit 1011.

Thereafter, the image post-processing unit 1013 extracts a depth map bycalculating a depth based on the disparity map computed at the stereomatching unit 1012. Then, an image post-processing is performed forsegmenting different objects from the extracted depth map to obtain theoutput image of the image post-processing unit (S5).

As a result, the horizontal and vertical lengths, and the distance ofthe objects segmented during the image post-processing operation (S5)from the robot terminal are extracted, whereby the movement and travelof the robot is controlled by segmenting/recognizing whether the objectsare objects to track or obstacles based on the extraction results. Ifnecessary, the robot may be controlled by information provided from therobot server 20. In addition, the image selection unit 1014 selectsanyone of the output images of each element of the vision processingunit 101 under certain circumstances (S7), and transmits the selectedimage to the robot server 20 (S8).

As described above, according to the present invention, the informationrequired for the robot's traveling and moving is processed in the robotterminal itself without transmitting the information to the robot serverby configuring the robot terminal with the stereo camera, and a lowprice dedicated chip and an embedded processor. Therefore, it ispossible to reduce the communication expense which may occur in case oftransmitting the necessary image information for driving the intelligentservice robot, so that the present invention is effective for embodyingthe network based intelligent service robot capable of providing userswith various functional services with low cost.

Furthermore, according to the present invention, since the acquiredimage can be used for extracting the three-dimensional distanceinformation of the external objects from the stereo camera as well asrecognizing the external circumstance by implementing the visionprocessing apparatus of the intelligent robot using the stereo camera,the present invention is effective for performing the intelligent robotto travel and move only with the stereo camera image processing withoutother sensors.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present invention. Thus,it is intended that the present invention covers the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

1. An apparatus for vision processing on a network based intelligentservice robot comprising: a stereo camera unit acquiring images from twocameras of left and right cameras incorporated therein; an input imagepre-processing unit for calibrating characteristic differences betweenthe left and right cameras with respect to the left and right imagesinputted through the stereo camera unit; a stereo matching unit forperforming a stereo matching by computing a disparity map through thestereo matching over the left and right images inputted through theinput image pre-processing unit; an image post-processing unit forextracting a depth map for the image outputted from the stereo matchingunit, and performing a segmentation for segmenting different objectsfrom the extracted depth map; and an image output selection unit forselectively outputting the output image of the respective units.
 2. Theapparatus of claim 1, wherein the input image pre-processing unit usescamera 20 calibration, scale-down of the image, rectification of theimage, brightness control of the image, or the like, in order that thestereo matching unit may perform the stereo matching over the imageinputted from the stereo camera unit with ease.
 3. The apparatus ofclaim 1 or 2, wherein the image output selection unit, which selects theoutput image of the vision processing unit of the intelligent servicerobot, selects and outputs a predetermined image selected from the groupconsisting of the image acquired from the left and right camera, theresultant image of the input image pre-processing unit, the resultantimage of the stereo matching unit, and the resultant image of the imagepost-processing unit, according to applications of the intelligentservice robot terminal.
 4. The apparatus of claim 3, wherein the imagepost-processing unit segments a target to track and an obstacle on thebasis of a three-dimensional image information computed through thestereo matching unit, and uses the horizontal, vertical lengths and thedistance of the segmented object as information for the robot'straveling and moving.
 5. The apparatus of claim 1, wherein the imageoutputted from the stereo matching unit is represented such thatdistance information is represented as a bright color for a close objectand as a light color for a distant object.
 6. A method for visionprocessing on a network based intelligent service robot, the methodcomprising: acquiring an input image of at least one of an object totrack and an obstacle from two cameras comprising a left camera and aright camera; performing an input image pre-processing to calibrate theleft and right image signals acquired from the left and right cameras tohave the at least one of the object to track and the obstacle, using atleast one or more image processing methods; performing a stereomatching, at a stereo matching unit, by computing a disparity map afterfinding out points corresponding to each other in the left and rightimages calibrated in an input image pre-processing unit; extracting adepth map by calculating a depth based on the disparity map computed atthe stereo matching unit, and performing an image post-processing forsegmenting different objects by the extracted depth map; and selectivelyextracting the horizontal length, the vertical length, and the distanceof the segmented object from at least one of the acquired input image,the performed input image pre-processing, the performed stereo matching,and the extracted depth map by calculating the depth based on thedisparity map computed at the stereo matching unit and the performedimage post-processing for segmenting the different objects by theextracted depth map of a robot terminal, and controlling the robot totravel and move by at least one of segmenting and recognizing the objectto track and the obstacle based on the selected extraction result. 7.The method of claim 6, wherein the controlling of the robot is performedby the robot terminal information provided by a robot server.
 8. Asystem using an apparatus for vision processing on a network basedintelligent service robot, comprising: a robot terminal including avision processing unit of a robot having the apparatus claimed in claim1 for acquiring an external image, a robot control unit for controllinga whole robot terminal, a robot server communication unit for serving arole of communicating with a robot server, and a robot sensor/drive unitfor sensing an external circumstance and driving the robot; and therobot server interfaced with the robot terminal over network, forcontrolling the robot under predetermined circumstances.